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SCIENTIFIC  INTEGRITY 


UNCLASSIFIED 


UNDERSTANDING 

Human  Judgment  in  Analysis 

•  How  can  we  take  advantage  of  human  judgment  in  a 
way  that  is  good  for  analysis? 

-  Human  beings  have  such  better  understanding  of  human 
contexts  than  computers  do 

-  In  analysis  like  irregular  warfare  that  involves  the  human 
terrain,  this  is  more  important 

•  The  DoD  does  not  trust  computer  simulation  in  this 
domain,  and  employs  wargaming 

-  But  it  is  difficult  to  get  enough  repetitions  for  statistical 
significance  of  human  judgments 

-  They  have  compromised  by  using  computer  wargame 
adjudicators  of  social  phenomena. 

•  PSOM,  SEAS,  COMPOEX,  etc. 


TECHNOLOGY 


Military  Operations  Research  Society 


i 

UNCLASSIFIED 


3 


SCIENTIFIC  INTEGRITY 


UNDERSTANDING 
TECHNOLOGY 


Computer  Adjudicators  for  Wargames 


Is  it  a  good  idea  to  use  a  computer  adjudicator  for  a 
wargame  for  analysis? 


-  You  could  get  this  worst  of  both  worlds... 

•  We  are  spending  lots  of  money  on  the  games  and  the 
software  but, 

•  You  wont  have  enough  repetitions  to  get  statistically 
significant  results  anyway 

•  The  computer  are  usually  better  at  playing  a  game,  on  its 
own  terms,  than  humans  are! 

-  The  world  champion  at  chess  is  a  computer  program 


UNCLASSIFIED 


•  Model-Game-Model  Technique 

-  Uses  iteration  between  human  and  computer  to  gain  the 
best  of  both  worlds 


-  Humans  are  used  to  improve  the  model,  not  replace  it 
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UNDERSTANDING 
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Model  Phase 


Model-Game-Model  Technique 


-  Explore  how  the  environment  in  the  game  may  be  manipulated 
so  that  an  agent  (human  or  ABS)  may  achieve  its  goal 

-  Bring  ways  to  “game  the  game”  are  to  the  surface 

-  Modelers  and  Subject  Matter  Experts  (SMEs)  may  change  the 
game  so  that  players  win  in  more  realistic  ways 

•  Game  Phase 


-  Players  play  the  improved  game 

-  Players  suggest  outcomes  that  are  more  realistic 

-  Modelers  may  change  the  game  so  that  players  win  in  more 
realistic  ways 

•  Model  Phase 

-  After  several  iterations,  realism  increases  to  the  point  that 
human  beings  are  not  needed  to  win  in  a  realistic  fashion,  at 
which  point  automation  (and  statistical  significance)  is  possible 
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UNDERSTANDING 

Strategic  Data  Farming 


•  Strategic  Data  Farming  may  be  employed  in  the  first 
phase  of  the  iterative  model-game-model  process 

-  Strategic  Data  Farming  is  a  way  to  explore  how  a  player 
or  an  agent  may  succeed  in  a  wargame  or  an  agent 
based  simulation  (ABS) 

•  Ways  to  game  the  game  are  exposed 


TECHNOLOGY 


•  Strategic  Data  Farming  makes  use  of  Game  tree 
technology  from  Artificial  Intelligence 


•  Strategic  Data  Farming  looks  at  worse-case-scenarios 
first 


-  Desirable  for  analysis 

-  Game  trees  win  by  the  exploration  of  the  worse  case 
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Why  do  ABS  and  Wargames  need  Strategic  Data 

Farming? 


ABS  and  wargames  are  typically  nonlinear 


-  New  combinations  of  parameters  contain  surprises 

-  Traditional  parameter  sensitivity  testing  for  VV&A  is 
inadequate 


•  Traditional  Data  Farming  does  a  thorough  exploration 
of  the  state  space 

-  Seeks  to  explore  every  combination  of  parameters 

-  Takes  supercomputers  and  vast  computational  resources 

•  In  Strategic  Data  Farming,  the  emphasis  is  on  the  game 
theory  of  moves  rather  than  parameters 

-  There  are  usually  fewer  parameters  in  strategic  games 

•  What  makes  a  game  unique  is  strategy 


-  Assumption  of  goals  narrows  down  what  exploration 
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Questions  Answered 


Traditional  Data  Farming  answers  the  basic  questions 
of  Agent-Based  W&A 

-  Is  every  model  outcome  possible  in  the  real  world? 

-  Is  every  possible  real  world  outcome  realizable  in  the 
model? 


•  Strategic  Data  Farming  answers  the  basic  questions  of 
strategic  games 

-  Do  strategies  that  win  in  the  simulation  win  in  the  real 
world? 

-  Do  strategies  that  win  in  the  real  world  win  in  the 
simulation? 


•  For  Strategic  Data  Farming,  once  the  model  is  refined 
so  that  the  answer  is  yes,  the  game  may  be  automated 
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How  Strategic  Data  Farming  Works 

•  Replace  a  player  or  an  agent  of  a  wargame  adjudicator 
or  agent  based  simulation  with  an  game  tree  agent 

•  The  game  tree  agent  needs  3  things 

-  A  board  evaluation  function:  a  way  to  tell  how  far  the 
agent  is  from  its  goal  for  a  particular  state  of  the  board 

-  A  list  of  all  the  legal  moves  that  the  player  can  make 

•  If  there  are  too  many,  they  should  be  ranked  according  to 
their  usefulness  for  a  strategy 

-  A  list  of  all  the  changes  that  can  be  made  in  the  agents 
environment 


TECHNOLOGY 


•  May  include  other  moves  of  the  players 

•  If  there  are  too  many,  they  should  be  ranked  according  to 
how  harmful  they  are  to  the  agent’s  strategy 

•  Invert  the  game 
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How  Game  Inversion  Works 


Play  the  game  as  though  there  are 
only  2  strategists 

-  The  replaced  agent 


-  The  rest  of  the  simulation 

•  A  game  tree  is  created  in  which  the 
rest  of  the  simulation  is  pitted  against 
the  agent’s  goals 

-  The  simulation  itself  is  used  to 
advance  to  the  next  move 

-  The  game  is  “branched”:  For 
every  move  made,  the  top  N 
moves  of  the  opponents  are  tried 

-  Alpha  beta  minimax  quickens  the 
search 

-  A  move  is  chosen  when  goal  is 
reached  or  computational  limits 
are  reached 
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•  Replace  every  agent  in  the 

simulation,  or  player  in  the  game,  in 
the  same  fashion 
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^^^0\Tiai“10W/iaf  Strategic  Data  Farming  Makes  Possible 


Increased  fidelity  of  wargames  and  agent  based 
simulations 


•  Replacing  the  players  for  full  automation 

-  Once  it  is  impossible  to  “game  the  game”  the  computer 
can  usually  play  the  game  better  than  a  human  can 

-  Human  creativity  doesn’t  matter  to  who  wins  game 

•  Imagining  meanings  for  chess  moves  never  won  the  game 

•  The  player  can  not  increase  the  ways  to  win  the  game,  he 
can  only  eliminate  consideration  of  non-human  ways  to  win 

-  If  the  problem  is  analysis,  this  will  save  human  resources 
and  make  runs  statistically  significant 

-  Only  applicable  to  closed  games  (That  only  machines 
adjudicate) 
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Myths  of  Game  Theory  Techniques 


A  game  tree  agent  makes  players  act  rationally  while  people  are 
irrational  in  the  real  world 


-  Game  tree  agents  evaluate  their  environment  according  to  their 
goals.  These  may  be  religious,  or  private 

•  Sometimes  people  just  miss  things  or  behave  stupidly,  but  we  are 
not  exploring  those  instances 

-  Game  tree  agents  do  not  need  to  take  poor  moves  into  account 
in  order  to  calculate  how  to  win  the  game 

•  Game  trees  can  not  handle  real  world  moves 


-  Modern  game  tree  technology  can 


•  Use  heuristics  to  rank  moves 

•  Take  into  account  simultaneous  moves 

•  Take  perception  into  account 

-  As  in  poker 

•  Take  probability  into  account 
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Summary 


•  Strategic  Data  Farming  can  perform  validation  and 
enable  automation  in  agent  based  simulations  and  in 
wargame  adjudicators 

•  Strategic  Data  Farming  narrows  the  space  of 
possibilities  that  need  exploration  in  strategic  games 

•  Strategic  Data  Farming  does  not  limit  human  creativity 
in  analysis  anymore  than  the  simulator  itself  does 


TECHNOLOGY 
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